AppIn No.: 10/790,218 Page 1 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



USER INTERFACE 
100 



BROWSER 
104 



VOICE • 
EXTENSION 
MODULE 
102 




APPLICATION 
SERVER 
130 



I 




107 
MOUSE 



T- KEYBOARD /^MICROPHONE 
MONITOR ^109 -.p . 

SPEAKER rlVJ. I 



VOICE EXTENSION MODULE 



PREPROCESSOR 
200 



PARSER 
202 



TRANSLATOR 
204 



SPEECH 
RECOGNITION 
ENGINE 
210 



INPUT HANDLER 
220 



FIG. 2 



Appin No.: 10/790,218 Page 2 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



^ START ^ 



3010 
A. 



USER 
INTERFACE 
INFORMATION 



3015 



RECEIVE USER 
INTERFACE 
INFORMATION 



3020 



PARSE USER 
INTERFACE 
INFORMATION 




3050 



PROCESS 

NEXT 
ELEMENT 



NO 



FIG. 3 



Appin No.: 10/790,218 Page 3 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



400 

A 



Q Craata Leave Raqueat - Micro5oft Intornel Exptorpf provided by SAWT 
' EDs fdt ^ So favoritea ije^ 



a A « 

Md Foito mm 



E II Linfci I 



[7- 



Quit Help 



SAP 



Create Leave Request 



Abence Typo: |Vacation Day 



410 



Absent From: 



Day 



i ^ -412 

To (indutive): 1 10/4^2001 I P N.^^^ ^ 



\ Vacation Balance "K .,,^^ ... 

^404 



Richafd Swan 



-416 



' "^ ^ 406 



1 



iastart I 



nnnnr 



I I jgpgdiie M....||aiDoci llBMytoiyulaf llWMjgpMftW-ll^^ | 



FIG. 4 



Appin No.: 10/790,218 Page 4 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



400 

L 

B Craata Loava Roquaat - Microaofl tntamat Exptowf pfOVldad by SAPIT 




G)« £dl ^ fio F»roritBS Uft|p 



Quit Help 



SAP 



Create Leave Request 



Absenca Type: 
Absent ftotn: 
To (inclusive): 

Approver 



Vacation Day: 



Half-DayVacation Day" 
Vacation Day: 



SAP Floating Holiday 
Half-Sick Day 
Sick Day 

Fam. Funeral Leave (3 day) 
Funeral leave (1 day) 
Jury Duty/WitnessSvo 
Wedding Day 

Birth of W{1 day) 



410 



I Vacation Balance ""K ^^^^ 



404 




406 

408 



innnr 



Ueal intrmt un» 



iflStartI 



D fjlC«llnalii.,||eiOoc» llgMyCompulifl 



FIG. 5 



Appin No.: 10/790,218 Page 5 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



( START ) 



602- 



DIVIDEUI 
ELEMENTS 
INTO GROUPS 



604- 



ASSIGN 
PRIORITIES 
TO GROUPS 



606 



RECEIVE 
VOICE 
INPUT 



DETERMINE 
GROUP 
MATCHING 
VOICE INPUT 




NO 



612 



614- 





YES 

r 


MARK 
POSSIBLE 
TARGETS 








RECEIVE 
VOICE 
INPUT 







618 

A 



IDENTIFY 
TARGET 
Ul ELEMENT 





FIG. 6 



Appin No.: 10/790,218 Page 6 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



J 



7myE5Bfcom 



a In Usa 



O Start Page 



OCATW 



B Worlylace Favorite* 



O City of a 



▼ VoWP ESS many 



QCATW 



T Cutoflpgr zinTofmation 



OZip Tach 
OJatCorp 
O Order information 



▼ Travel Aqant 



OAirflnea 
O Rental Car* 
O Vacation Plannar 



▼ Generic services"" 



OSent documents 
OSand object with note 
OWorfcflow overview 
OPrivata notes 
OUnks 

QStart workflow 



400 

A 



mySAP.com 



SAP 



Welcome demo CRC 



704 



II iijiQcai.ninasr-n 



FIG. 7 



TmyEABfcom 



QCATW 



O aiyof* 



B In Use 



O Start Paga 



B Worfcplaoa Favoritaa ^ 



B Rotes 



▼ VbWP ESS menu 



QCATW 
r Customer imonnaBofT" 



OZip tech 
QJetCorp 
O Order Information 

Y Travel Agent 



QAiriinas 
Q Rental Cars 
O Vacation Planner 
V Gene ric servioes 



OSent documents 
QSend object with nots 
Q WortcfkMM over>riaw 
O Private notea 
QUnks 

O Start %n»1(fkiw 



TT7 



im 



700 



Quit Help 

Time Sheet 



NanM 


Mr. Jeny Wagner | 


rWSOnMI NO. 


1S113J 




SAP 



Entry period 


Mr.|O1/14/20O2 |to |01/20/2002 


iPrevious | [Next] 






[ 01/2(V2002 1 



Data entiy pronia 



lESSflWKFl I 



Att/sbs.^ 



Tygat limes 



FR 
01/18/ 



llnwrtline | | Shew atl detaliri | Rint | |Copy prevkius period | 
Thi« It your time sheet tor the current debt snky period 



806 



k entries 1 ISave 



ji >i 11.^ Locsii;?:^' 



FIG. 8 



Appin No.: 10/790.218 Page 7 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



TmyES^nm 



700 

A 



Time Sheet 




SAP 



702- 



i InUM 



Q St»t Page 



O CATW 



B Wor>cplaoo Favorites 



O City of « 
Roles 



▼ VoWP ESS menu 



Q CATW 
y CuBtotner Infoonation 



O ZipTach 
Q JotCorp 
0 Order Information 



T Travel Agent 



OAiilines 
ORental Cars 
QVacation Plarwer 
W Generic aarvteaa 



OSent documents 
QSend obiect with note 
9 Workflow ovarvi 
a Private notes 
O Links 

O Start workfkiw 



XE 



Nam 


h^. Jerry Wagner | 


Parwnndno. 


11113 1 



Entry penod 



[01/14^002 I |01/t4/2OO2 [to t)t/12tt002| 
lootolhi 



Data entry profiia |ESSWKFL 



[rotal ojy^/ o^^s/ pjfjffe/ 01^7 oj)ia )j/19 01/2C 



Tvgattli 



Attsndanoe Hotn 



AOsndano* Hous 



AttmdmoeHovs 



IZ 



IZ 



IZ 



m. 



12 



-902 



pnsertlinel jShow all detaiij jPnnt | |Copy previous perioc^ 
This is your time sheet for lha cuinnt data entry period 



7t 



806 



I Checfcentries I ^we 



3 



FIG.9 



700 

A 




702- 



O start Page 


OCATW 




X 


a Wolkplaoa Favoritaa 




O Cityofe 




B Roles 




▼ VoWP ESS menu 




OCATW 




▼ Customer Information 




O Zip Tech 
OJotCorp 
OOrder Informalton 




T Travel Aoant 




OAiriines 
Q Rental Cars 
Vacation Planner 




▼ Generic ServiCM 




OSent documents 
O Send ofaieot with note 
OWorkflow overviaw 
O Privets notss 
OUnks 

OStBrtworfctow 








*l 1 







Quit Help 



Time Sheet 



Name 


M^. Jerry Wagner | 


mmnM no. 


1S113 1 



Entry Parted [01/14/2002 | to |01/20/2002 



Time Sheet ^^'^ 



1002 802 



OataerttiY profile lESSWKFL | 



|01/20/2002 


[PrvMoua 1 [Next 1 




Goto 1 i 01/20/2000 1 



Target Times 



Attendance Hours 



Attendance Hours 



Attendsftoe Hours 



Atlondanoe Hours 



MO TU WE TH FR SA SU 
01/14/ OinV 01/18/ 01/17/ 01/16/ )1/19/ 01/20y 



Insert inel | ShofsBdetailil |Print | | Copy prewcus pedod | 
This la your,tiina iheat fbr 0w Gunrant data anby psfkid 



806 



Locai intranet 



FIG. 10 



Appin No.: 10/790,218 Page 8 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



Q CATW 



B In Usa 



O Start Pago 



a Workplace Favoritea 



O City of e 



B Rolea 



' VoWP ESS menu 



rcui 



CATW 

istomer Information " 



OZip Tech 
OJetCorp 
QOrdor Information 



T Travel Agent 



OAini 

O Rental Cora 
(^Vacation Ranner 



y Generic servioea 



OSent documenta 
OSend object with noti 
QWorttflow overview 
O Private notaa 
Ol-inka 

O Start workflow 



XE 



700 
A 



Quft Help 

Time Sheet 



Name 


kikJenyVlfagnar 1 


Panoiffwiiio. 


18113 1 




Entry period 



Data erttiY protHe 



I ESSWXFL I 



|01/2(V2(»2 


iPrawioua 1 jNextl 




QolD 1 1 01/20/2002 1 



!7" 



Att/aba.type Total ^^4^ pjiVs/ OViV 01/18/ 0^^9)0?^^ 



Target limea 



Atlandanoe Houra » 



Attendance houra 



Attendance Hours w 



Attendance Hours v 



llnaertinel IShow aB detail 1 I Print | [ Copy piewious period 
This is your time aheet for Sw currant data antoy period 



7^ 



806 



I Check entrieal gave 



FIG. 11 



" mylSRBfcom 



0 start Page 


0 CATW 




X 


B Wor1(.plaoe Favorites 




0 City of e 




B Rolea 




▼ VbWP ESS menu 




OCATW 




▼ Customer Information 




OZip Tech 
0 J«tCorp 
0 Order information 




▼ Travel Aoent 




0 Airlinea 

0 Rental cars 

0 >taoation Planner 




y Generic aervicea 




OSent documents 
0 Send object with note 
0 Wortdlow ovofviaw 
0 Private notaa 
SUnka 

0 Start workfkiw 















myiSAHCom 



700 

A 



Quit Help 

Time Sheet 



NMie 


Mr. Jerry Wagner | 


Peraorwwlna 


ieii3 j 



902 




SAP 



Erttry ponod 



Data entry prolilo I ESSWKFL j 



101/20/2002 


I PrevkxisI |Next | 




Goto t 101/20/2002 1 



lAtt^^type |rotal 


MO 
01/14/ 


01/1 &/ 


oi^i^ 


TH 
01/17/ 


FR 
01/18/ 


SA 

}1/19/ 


SU 

oi/2a 






^ 


^ 










li;^ 


Taniet times 


40 


B 


'8 


8 


8 


8 






Attendance Hours 








8 






































^A 


Attendnoe Hours 
























Attsndanoe Hours 






















pTbtal 


0 


0 















llnaert lin^ Ishow aU detalij I Print | jCopy previous periodi 
Thia ia your time aheat for lha current data enry period 



806 



I Check entriaa 1 |save 



FIG. 12 



Appin No.: 10/790,218 Page 9 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



700 
A 




-myBABtom 



Qiit Holp 



Time Sheet 



702 



I In Use 



Q Start Page 



O CATW 



B Wofkrtace Favorite» 



IS 



^'Cityofe 



▼ VdWP ess menu 



CATW 



T fiuittnmflr InTni 



O Zip Tech 
OJetCorp 
QOider infofmaiton 



T TmYBl ftnont 



OAiriines 
O Rontal oare 
O Vacation Ptanner 



O Sent documents 
OSend obiectwith note 
O WorMlow overview 
O Private notes 
SUnke 

O Start woritflow 



JJL 



None 


Mr. Jwry Vbg'^er | 


Pmwxrmi no. 


18113 1 



Entry pfiod 



1 01A0Q002 


{Previous 1 {Noxtl 




Goto 1 1 01/20/2002 | 



Annual Ihm 
/Utondanoa Hous 

was Trip 
Compansstion Fisx Thw 



Education /TraMng 
FnigyMMlcil Leave 

Holiday 
Jury Duty 

LongTannDiaiMfty 

PTOFbAg Holiday 
Savaranoe Pay 
Short TafmPUabiity 





Total 


MO 

01/14/ 


TU 
01/15/ 


WE 
01/1fi/ 


TH 
01/17/ 


FR 
01/18/ 


SA 

91/19/ 


SU 
01/20/ 




> 


> 






> 








40 


8 


8 


8 


8 


8 










































































w 


























0 





















llnaartlinei | Slwwal<iataib1 IRW I | Copy previouapariod | 
Th» b your 6ne ahaat for the Gurant data anty pariod 



7^ 



806 



[ ChackantiMl jSa»a 



FIG. 13 



TOO 

A 




Quit Halp 



Time Sheet 



702- 



O Stwt Page 



OCATW 



BWoricpleoe Favorites 



O City of e 



▼ \toWP ESS menu 



QCATW 
T Cuatomar Infomriation 



O Zip Tech 
O JotCorp 
O Order infomiation 



V Tfavel Agent 



tVacation Planner 
V Generic aervlcea ~ 



OSentd 
O Send otiiwt with note 
O Woricflow overview 
O Private rratss 

Q Start worltflow 



NanM 


k^. Jarry Wagnar | 


Paraonnalno. 


18113 i 



EnfrVpariod 



|01/20«002 


1 Prawkxjaf [Next 1 




Goto 1 1 01/20/2002 | 



Att/afae.type 



T«.- WO TU WE TH FR SA SU. 
01/14/ 01/15/ 01/18/ 01/17/ 01/18/ J1/19/ 81/20/ 



Target times 



Attendance Hours 



Attandanoe hours 



Attendance Hours « 



Attftodanoo Hours 



^naert line| |Show all detaial | Print] jCopy previoua period] 
Thia is your time sheet for the current date enry period 



7^ 



806 



3 ^ 



As 



mySABtom 



FIG. 14 



Appin No.: 10/790,218 Page 10 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



700 

A 



Qjit H«lp 

Time Sheet 



804 

5 



Q Start Paew 



»CATW 



B Woffcplaoe FavoritM 



3 City of* 



O Role* 



yVoWP ESS menu 



PCATW 



Y Customor Information 



OZipToch 

QOrder information 
^ Travel Agent 



Q Rental cars 
OVacation Planner 



y Generic sarvioe* 
OSent documentB 
QSand object with note 
0Worl(flow overview 
O Private notea 
gUnlu 

QSIartWoiltflow 



Nana 


Mr.jMiyW^nar 1 


Peraonnolno. 


18113 1 



4- 



Entv Patted I 01/14/2002 | to | OlflOOOoTI iPreyouT] fTtot 



Dalaanlrypfe«a I ESSWKFLj 





Att/atxtypn 


Total 


uo 

01/14/ 


TU 

01/15/ 


WE 

01/16/ 


■m 

01/17/ 


FR 

01/18/ 


SA 
11/18/ 


SU 
01/20 








> 






> 












e 


8 


e 


8 


8 








AtlenderiM Houra 






40 






V 


5/ 










Holiday 








f 
















ARandme Hour* 












14/ 




v. 








Ationdanoe Houra 


• 






if 




19/ 


20/ 


21/ 

























I Insert inel | Show aB detal*~l I Print] | Copy piwlotis period | 
this is your tiina sheet for the currant data period 



m 



FIG. 15 



700 

A. 




702- 



OCity of e 
B Roles 



a In Use 



O Start Paao 



► CATW 



B Workplace Favorites 



pCATW 



ustonner InformatioTT" 

§Zip Tech 
JetCofp 
Order information 



▼ Travel Aoont 



QAirtinea 
O Rental cars 
O Vacation Planner 



O Sent docunwnta 
OSend objact wrilh not 
OWori(flow overview 
O Private notes 
O Unks 

O Start workflow 



Quit Help 



Time Sheet 



Name 


Jerry Wagner | 


Psraofwielno. 


18113 1 



Erittv period | 01/140002 | to | 01/20/2002 



iftevious I 



1602 802 



Dataa^^nfil^ 



Goto I I 01/20/2002 



2002 I 



i7" 



fS^J MO 
01/14/ 



AUsrvlanoe Hours 



AUondsfKa Hcxvs 



WE 
01/16/ 



TM 

01/17/ 



FR 
01/18/ 



01/19 01/20/ 



SA 



SU 



I Insert Ifcie | [ Show ^1 dstsaTH [Pri"! I | Copy previous period 
This is yourlim* sheet for the ourrent dais snry psrtod 



7^ 



IChecfcenWes "1 I Saw 



II HJLocJ^ 



FIG. 16 



Appin No.: 10/790.218 Page 11 of 14 

Applicant(s): Frances James ^^.^k, 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 

ELEMENTS | 

I 

I 



O CATW 



O City of • 



myKSESom 



O start Page 



□ Workplooo Favoritoa 



B Rolea 



VVfoWP ESS rrwnu 



QCATW 



yTfBwl Agent 



Zip Tech 
JatCorp 

fOrder information 



Vecation Planner 



V Generic aeivicea 



O Sent documents 
OSend object with noti 
O Wortcflofw overview 
O Prtvato notes 
O LJnite 

O Start woildlow 



m 



700 

A 



Quit Help 

Time Sheet 



Name 


Mr. Jsny Wagner | 


Pwiorawliio. 


18113J 




SAP 



Data entry profile | ESSWKFL | 



[01/20/2002 


jPrevious | |Next 1 




Goto 1 1 01/2W2002 1 



Tlnw Sheet ^^^^ 



ABwtdvm Hours 



UO 

01/14/ 



TU 
01/15/ 



WE 
01/16/ 



|rn«artlne | |SN>wildetril» H |P»»nt | | Copy pieww period 
This )« your lirri^wal lor thtouiranl data «f*ypwM 



1702 




806 



ICtwdiwIrias "I |Sw> 



FIG. 17 



7 mySABtom 



O CATW 



O atyofe 



i In Use 



O Start Page 



"TP? 



a Worttplaoe Favorites 



B Roles 



▼ VoWP ESS menu 



QCATW 
y Cuatofftef Information' 



O Zip Tech 

SJetCorp 
Order Information 



y IraveiAgenIt 



§Airiines 
Rental cars 
Vacat' 
^Cenerto 



O Sent documents 
OSend oliiect with note 
O Worlcflow overview 
2 Private notes 
O Unk« 

O start worl^ftow 



ML 



700 

A 



Qjit Help 

Time Sheet 



Name 


|y^. Jerry W^r| 


Psraomel no. 


18113 1 



6 Shad 



806^ 802 




1 01/20/2002 


1 Prevnus] |Next 1 




Goto] 1 01/20/2002 | 



Data entry profile IeSSWKFL | 





Att/abs-type 


liDtel 


MO 
01/14/ 




TU 

oi/iy 




WE 
01/16/ 
















From 


To 




w 


From 


To 






From 


To 






Target times 


40 


0 










08:00 


17:00 






08:00 


17:00 






AttorxJanoe Hours 






8 








9 








8 










Holiday 














3 


















Attendance Hours 
































AttBrujarK» Hours 






























To 


tal 




0 

























tnsertHnj iHkle all details I [Wrt] [Copy previous periocj 



ItfLooail, 



|5Ss 



FIG. 18 



Appin No.: 10/790,218 Page 12 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



1900 



1906 
A. 



VOICE-ENABLED 
USER INTERFACE 



OPEN INTERACTION 
ELEMENT 
1910 



INTERFACE 
ELEMENT 1 
1908 



INTERFACE 
ELEMENT2 
1908 



VOICE MODULE 
1912 



DATA ENTRY 
MODE 
1914 



1922- 



NAVIGATIONMODE 
1916 



1000 COMMAND GRAMMAR 


.VOICE TABS 




EXPLICITEXIT ^ 
COMMANDS 






NAVIGATION 
' COMMANDS 




ALL ELEMENTS 
COMMAND 


TIMEOUT 




/ 
1930 


IMPLICITEXIT 
COMMANDS ^ 



SPEECH RECOGNITION ENGINE 
1918 



MICROPHONE 
1904 



COMPUTER N 
1902 



FIG. 19 



Appin No.: 10/790,218 Page 13 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



2000 



OUTPUTAN OPEN INTERACTION ELEMENT 
INAVOICE-ENABLEO USER INTERFACE 



-2002 




ENABLE EXPLICIT 
EXn OPTION 



T 

2004 



ENABLEIMPLIGIT 
EXITOPTION 



ENABLE 
VOICE TABS. 



START TIMEOUT 
PERIOD 




RECEIVE DATAFOR ENTRY IN 
THE OPEN INTERACTION ELEMENT 



-2012 



UPDATE THE OPEN INTERACTION 
ELEMENT WITH THE RECae DATA 




■2014 



DETERMINE EXIT 
OPTION SELECTION 




EXIT OPEN 
INTERACTION 
ELEMENT 



-2028 



FIG. 20 



Appin No.: 10/790,218 Page 14 of 14 

Applicant(s): Frances James 

NAVIGATION AND DATA ENTRY FOR OPEN INTERACTION 
ELEMENTS 



2100 



FIRST NAME: 



ORDER NUMBER: 



2104 



LAST NAME: 



K 



2108 



T 
2106 



'2102 



ITEM1: 



(Please select item) 



2112 



REASON FOR RETURN: 



DESCRIPTION OF PROBLEM: 



wrong size 
wrong color 
wrong product 
arrived late 
defective/broken 
other - 




ITEM 2: I (Please select Item)' | 



REASON FOR RETURN: 



DESCRIPTION OF PROBLEM: 



wrong size 
wrong color 
wrong product 
arrived late 
defective/broken 
other 



'2110 




2118 



A. 



SAVE 



CLEAR 



-2120 



FIG. 21 



